There is Individualized Treatment. Why Not Individualized Inference?

Doctors use statistics to advance medical knowledge; we use a medical analogy to introduce statistical inference “from scratch” and to highlight an improvement. Your doctor, perhaps implicitly, predicts the effectiveness of a treatment for you based on its performance in a clinical trial; the trial patients serve as controls for you. The same logic underpins statistical inference: to identify the best statistical procedure to use for a problem, we simulate a set of control problems and evaluate candidate procedures on the controls. Now for the improvement: recent interest in personalized/individualized medicine stems from the recognition that some clinical trial patients are better controls for you than others. Therefore, treatment decisions for you should depend only on a subset of relevant patients. Individualized statistical inference implements this idea for control problems (rather than patients). Its potential for improving data analysis matches personalized medicine’s for improving healthcare. The central issue—for both individualized medicine and individualized inference—is how to make the right relevance robustness trade-off: if we exercise too much judgement in determining which controls are relevant, our inferences will not be robust. How much is too much? We argue that the unknown answer is the Holy Grail of statistical inference. Prologue: The Data Doctor A usual course on statistical inference teaches its mechanics: how to construct confidence intervals or conduct hypothesis tests through the use of probabilistic calculations. We then become so fluent in the modern language of statistics that we forget to ask: Why did the language develop this way? Can inference be done using a different language? For example, why introduce the notion of probability at all—is it truly indispensable? This article does not aim to introduce the reader to the language of inference as currently spoken by many statisticians. Instead, we try to create a language for inference “from scratch,” and stumble upon the main statistical dialects (and a few variants) by accident. By from scratch, we mean that we justify each step in the construction employing only “common sense”. In fact, we claim that one only needs to understand the following problem to understand statistical inference. The Doctor’s Problem. A doctor needs to choose a treatment for her patient, Mr. Payne. Her options are the standard treatment, A, and an experimental treatment, B. What does the doctor do to make her decision? She goes and finds out how A and B worked on other patients. Suppose she discovers that treatment B outperformed A on patients in a large randomized clinical trial. She wants to apply this result to Mr. Payne. For this to be a reasonable action, the patients in the clinical trial need to be good controls for Mr. Payne (who is 50 years old, weighs 200 pounds, exercises once a month, etc.). Of course, she realizes that not all patients in the trial are good controls (certainly not the 12 year old girl). So she selects a subset of patients